The two tests in SearchableSnapshotsPersistentCacheIntegTests 
failed few times when they were expecting the persistent cache to 
be empty after mounted indices deletions.

While I wasn't able to reproduce the issue (which happened ~5 
times in the last 2 months) I suspect the cause are the same: the 
tests wait for the mounted index to be green but don't always wait 
for the prewarming to be completed. I think it is possible that a 
part of a cache file finishes to be prewarmed after the test is 
completed, making the next step to fail as the persistent cache 
will contain a doc that is not related to the subsequent test.

This commit changes the testCacheSurviveRestart so that it 
waits for the prewarming to complete. Similarly it changes t
estPersistentCacheCleanUpAfterRelocation to also wait for 
the prewarming to complete (just in case, this test verifies that 
the recovery is done). It also logs more information about the 
emptiness of the persistent cache on data nodes and makes 
the cluster scope to TEST to ensure that a dedicated test 
cluster is used for each test.

Closes elastic#76159
Closes elastic#76160